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Abstract 


A number of national and international organizations have compiled lists of grand challenges to unify 
the efforts of scholars and practitioners in a field. Unified efforts increase the possibility of creating 
meaningful and lasting progress. In this paper we share ten grand challenges that were identified through 
an examination of the assessment literature and a national survey. Each of the grand challenges are 
described. The four challenges which were identified as being of greatest concern to the assessment 
community are currently being addressed by the Grand Challenges in Assessment Project for possible 
collective solutions. 


National Institute for Learning Outcomes Assessment 


Grand Challenges in Assessment: 


Collective Issues in Need of Solutions 


Karen Singer-Freeman & Christine Robinson 


A grand challenge is defined as a problem requiring collective cooperation within a 

community of scholars to successfully resolve. Grand challenges were first identified via a 

publication of a list of mathematics problems by Hilbert (1902) in an attempt to advance ——_—_—_—_—_—— 
creative solutions. Since that inception, grand challenges have been identified by national 

and international organizations who generate lists of challenges to unify and converge 

the efforts of scholars and practitioners within colleges and universities and affliated Grand challenges 
organizations (Omenn, 2006). While grand challenges have been undertaken to address gyfyfyess enduring issues 
problems such as creating economical sources of solar energy (National Academy of , : 
Engineering, 2016), developing renewable fuel alternatives (National Research Council, in need of. solution 
2005), and including active science inquiry in all introductory college science classes through collective 
(Alberts, 2013), the field of assessment had not yet employed the concept of grand : 
challenges to address some of the most enduring issues in need of solutions through research and action. 
collective research and action. This paper explores why grand challenges are a useful Thijs paper presents ten 
conceptual framework to advance assessment efforts and outlines the ten grand challenges . 
identified in a review of the literature—arguing that it is indeed the case that the field grand ch allenges in 

of assessment in higher education has grand challenges in need of collective solution gesessyyent. 

development.' 


Grand Challenges and Assessment 


Perceptions about the value and worth of assessment in higher education at times are 
bleak. A survey of chief academic officers found that almost a third thought assessment was 
for appeasing politicians and accreditors as opposed to improving teaching and learning, 
while almost a fifth did not agree that systems of assessment have led to improvements in 
quality of teaching and learning (Jaschik & Lederman, 2020). This is not a new narrative 
to assessment professionals who are well aware of the negative perceptions bestowed 
upon assessment, and frequently struggle to assuage negative perceptions of assessment 
(Ariovich et al., 2019). However, as is often the case with wicked and complex issues, 
simply ignoring them will not make them go away. 


To identify the grand challenges of assessment in higher education, we reviewed assessment 
websites, blogs, discussion boards and publications from 2015-2019. We chose to limit 
the review to this time period to maintain a future oriented perspective. The reviewed 
materials included 83 pieces of writing with 46 non-peer-reviewed sources, 34 peer- 
reviewed sources, and three blog or discussion board posts. This review resulted in the 
identification of ten potential challenges that were frequently mentioned and fulfilled the 
four defining characteristics of grand challenges: 


Extremely hard to do, yet doable. 

Produces positive outcomes potentially affecting large numbers of people. 
Associated with clear metrics and goals so progress and completion can be identified. 
. Captures popular imagination, and thus garners political support (Gould, 2010; 


Stephan et al., 2015). 


"To learn about the full process of identifying and narrowing the grand challenges for assessment see 
Singer-Freeman and Robinson (2020). 


AYN 
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These challenges were included in a national survey of higher education assessment 
professionals (Singer-Freeman & Robinson, 2020). The four challenges with broad 
support now serve as a starting point for strategic planning and collective problem solving, 
an effort currently underway, with endorsements from nine national organizations 
(American College Personnel Association, Indiana University-Purdue University 
Indianapolis Assessment Institute, Association for Institutional Research, Association for 
the Assessment of Learning in Higher Education, Association of American Colleges and 
Universities, Council for the Advancement of Standards in Higher Education, NASPA: 
Association of Student Personnel Administrators, National Institute for Learning 
Outcomes Assessment, and Student Affairs Assessment Leaders). In this paper we describe 
all ten challenges and invite assessment professionals to begin the important work of 
creating solutions. 


The Ten Grand Challenges in Assessment 


Drive Innovation. The process of assessment should produce visible and actionable 
assessment findings that drive innovation. Innovation requires future-oriented 
considerations of change that brings together assessment and planning activities 
(Jorgensen, 2018). The field of assessment has shifted from conducting assessment to 
demonstrate compliance towards producing actionable assessment findings to drive 
learning improvement and informed decision making (Baer, 2017; Blaich & Wise, 2018; 
Horst & Ames, 2018; Ikenberry & Kuh, 2015; Jankowski, 2018; Kuh et al., 2015; 
Pasquerella, 2018; Roscoe, 2017; Stanny, 2018a, 2018b; Suskie, 2019). Although there 
has been an increase in the use of data-driven decision making, there is not evidence that 
data-driven decisions have improved students experiences or outcomes (Cox et al., 2017). 
This finding might be explained by a lingering challenge. To effectively drive innovation, 
it is essential to improve assessment methodology so that evidence gathered informs our 
understanding of the outcomes associated with innovative practices. In addition, we 
must identify the causes of gaps in student learning, identify evidence-based solutions 
from the research literature, determine whether selected interventions are implemented 
with high fidelity, and measure the extent to which the interventions drive learning 
improvements (Eubanks, 2017; Fulcher et al., 2017; Smith et al., 2017; Stitt-Bergh et 
al., 2018). Assessments that drive innovation also require authentic faculty involvement 
(Neuschel & Rego, 2018; Rickards & Stitt-Bergh, 2016; Roscoe, 2017; Stevenson et al., 
2017; Suskie, 2015). As one of the four challenges with broad support, this challenge 
is currently being addressed by the Grand Challenges in Assessment Project (Singer- 
Freeman & Robinson, 2020). 


Inform Budget. Assessment findings should be used to inform budgetary decisions. As 
the cost of higher education rises precipitously, there is increasing urgency in finding 
ways to reduce institutional costs and direct funds strategically for improvement (Kuh 
& Ikenberry, 2018; Suskie, 2016). Performance-based funding for state colleges and 
universities has been identified as a top issue facing higher education by the American 


Association of State Colleges and Universities and has been recommended as a best practice 
by both the Lumina and Gates Foundations (Baer, 2017). Return on investment tools 
estimate the extent to which instructional improvement efforts (Rossman et al., 2019) 
and student success initiatives (Desrochers & Staisloff, 2018) are likely to provide value 
beyond their predicted costs. Tools like these have the potential to integrate assessment 
findings with research findings. Nonetheless, inherent in performance-based funding is 
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the tendency for funds to flow towards current success which may conflict with long term 
strategic priorities. For example, performance-based funding might negatively impact 
equitable access to higher education if institutions only admit students who are most 
likely to graduate based on previous success rates. To maintain a focus on student success, 
it is important that we increase the extent to which we effectively assess progress towards 
strategic objectives and establish performance metrics that are in alignment with strategic 
goals. One element of a successful approach to this challenge is ensuring that sufficient 
funds are set aside to support high quality assessment activities and appropriate levels ee 
of staffing in offices of assessment (Ewell & Ikenberry, 2015). High quality assessment 
findings must be communicated with those responsible for budget decisions in order to We encourage 
successfully align assessment findings and budget allocations. We encourage institutions + eteutions to take 
to take steps towards increasing the integration of planning, assessment, and budgeting. 

steps towards increasing 
Immediate Improvements. Assessment findings should be used to direct immediate hp Rape erat dn 2 off 
pedagogical improvements. Too often, assessment findings are not utilized to direct 
immediate pedagogical improvements, in part because the work of closing the loop in planning, assessment, 
student learning outcomes assessment is too slow to benefit the students who are assessed 
or to improve the instruction or course design of those who are teaching (Eubanks, 2017; 
Maki, 2017). We must find ways to make changes in response to assessment findings 
within the space of a single class through formative assessments (Dirlam, 2017; Lépez- 
Pastor & Sicilia-Camacho, 2017; Maki, 2017). The rapid increase in online teaching 
and adaptive learning provides opportunities for assessments to take place in real time 
and may result in a shift to individualized instruction (Deeley, 2018; Neuman, 2017). 
Integration of information about student cognitive skills, social-emotional development, 
and current academic accomplishments can now be provided rapidly to faculty and 
students (Baer, 2017). This information can be used to improve pedagogy by providing 
faculty with information about how their current pedagogy is impacting individual 
students. If technology provides timely data to students and faculty, these data can support 
the rapid delivery of interventions to enhance and support student success (Baer, 2017; 
Shacklock, 2016). To measure the success of immediate pedagogical improvements, it 
will be important to measure student learning over time and encourage students to 
reflect on their own learning which might involve expanding the use of ePortfolios to 
encourage student reflections. As one of the four challenges with broad support, this 
challenge is currently being addressed by the Grand Challenges in Assessment Project 
(Singer-Freeman & Robinson, 2020). 


and budgeting. 


Increase Equity. Assessment findings should be used to increase educational equity. 
A goal in higher education is that every student has an equal opportunity to succeed 
regardless of ethnicity, gender, socioeconomic status, ability, or family educational history. 
There is compelling evidence that we are not meeting this goal (Cahalan et al., 2018). 
We must design and analyze assessments to reveal the extent to which institutions of 
higher education are providing access to high quality education for all students (Gavin 
et al., 2018; Jankowski, et al., 2018; Klonoski et al., 2018; Kuh & Ikenberry, 2018; 
Nunez, 2018; Pasquerella, 2018). There has been increasing attention given to the role of 
assessment choices in perpetuating equity gaps (Montenegro & Jankowski, 2017a; 2017b; 
2020; Singer-Freeman & Bastone, 2019a; Singer-Freeman et al., 2019) and the role that 
assessment activities play in supporting educational equity in higher education (Blaich 
& Wise, 2018). When we identify educational inequities, we must carefully examine the 
effects of current practices on underserved groups, viewing the gaps as resulting from 
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failures of practice rather than students’ problems (Malcom-Piqueux, 2018). We must 
also consider ways to make the assessment process itself fair. To effectively increase equity 
in higher education, we must increase our use of data disaggregation. As the challenge 
with the broadest support, the challenge of increasing equity is currently being addressed 
by the Grand Challenges in Assessment Project (Singer-Freeman & Robinson, 2020). 


Disaggregate Data. Data on learning should be disaggregated to consider important 
student characteristics. For the most part, assessment data are reported and reviewed 
in ways that mask inequities because student learning outcome reports aggregate 
assessment results across all sections of courses and instructors. To reveal the extent to 
which institutions of higher education are providing equal education to all students, we 
must examine disaggregated outcome results (Garrigan et al., 2018; Gavin et al., 2018; 
Harper et al., 2018; Klonoski et al., 2018; Singer-Freeman et al., 2019). Yet, a NILOA 
survey of provosts revealed that the use of assessment data to improve educational equity 
in higher education is uncommon (Jankowski et al., 2018). Disaggregation along with 
the consideration of other variables, including previous educational experiences, can also 
be used to assess the extent to which curricular and co-curricular experiences support 
student success (Brown, 2017; McNair, 2018). As we increase the use of disaggregation 
practices, we must carefully consider which groupings of students are appropriate. For 
example, we lump a large number of ethnicities into the group “Asian American.” When 
groupings are too broad, we cannot assume that what is revealed is true for everyone in 
that group. Maximally useful data disaggregation would be supported by the availability 
of large data sets. We encourage institutions to increase the disaggregation of student 
learning outcome assessment results to reveal whether the attainment of learning is 
equitable across the institution. 


Change Over Time. To identify progress, it is essential to examine changes in institutional 
effectiveness (including student learning) over time. Strategic planning in business 
effectively supports continuous improvement because of rigorous follow-up which 
includes monitoring progress towards goals, attending to changes in market conditions, 
and responding by resetting tactics (Gordon & Fischer, 2015). Unfortunately, higher 
education strategic planning can be ineffective at driving improvement because of limited 
follow-up (Gordon & Fischer, 2015). Gordon and Fischer (2015) found that in higher 
education, strategic planning is frequently viewed as a task that must be completed to 
meet accreditation standards and that strategic plans are often viewed as a means of 
communicating with an external audience rather than as realistic plans for future activities. 
To maximize the usefulness of strategic plans, there must be meaningful tracking of 
progress towards institutional effectiveness goals over time (Harvey, 2017; Jorgensen, 
2018; Suskie, 2015) and linkages between strategic planning and budget allocations. 
One key element of tracking institutional effectiveness is the measurement of individual 
students’ learning (Eubanks, 2019a; 2019b; Hundley, 2019; Jankowski & Marshall, 
2017). This might include tracking of long-term outcomes-based performance measures 
including successfully paying off student loans, post-graduation earnings, and research or 
innovation that benefits society (Baer, 2017; Miller, 2016; Pasquerella, 2018; Rickards 
& Stitt-Bergh, 2016). If we are to track learning over time effectively, we must increase 
the use of technology that provides longitudinal student data. This might involve the 
expanded use of ePortfolios that document student learning over time or the use of more 
sophisticated databases and analyses. As one of the four challenges with broad support, 
this challenge is currently being addressed by the Grand Challenges in Assessment Project 
(Singer-Freeman & Robinson, 2020). 
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Student Self-Evaluation. Involving students in authentic self-evaluation of their own 
learning enhances the learning process. A key element of student learning that may 
predict students’ long-term success is their capacity to learn (Boud & Soler, 2016). To 
develop a capacity for learning, it is helpful for students to become explicitly aware of their 
learning (Havnes & Proitz, 2016). Students who engage in self-assessments become more 
aware of their own learning and feel a sense of ownership over this learning. Additionally, 
students’ assessment of their own learning provide rich evidence of learning that can 
be used for improvement (Bourke, 2018; Lépez-Pastor & Sicilia-Camacho, 2017; Lurie a 
& Garrett, 2017; Zeng et al., 2018). Competency-based education or ePortfolios are 

ways to encourage students to evaluate their learning and increase their awareness of We encourage 
accomplishments (Singer-Freeman & Bastone, 2019b). Dann (2014) links the use of self- 
assessment activities to increased curiosity, awareness of learning gaps, and self- regulation 
around educational efforts. A new model of the student as a partner and change agent in 2/Credase the use 
sean has emerged (Healey et al., 2014) in which student self-reflections improve off elDae: tfo Vane 

our understanding of more traditional assessment results. To fully integrate self-evaluation 

of learning into pedagogy, it will also be important to provide faculty with training on provide students with 
best practices in evaluative student reflections. Increasing student self-evaluations has the 


institutions to 


potential to improve both student learning and the assessment of student learning. We documentation of 


encourage institutions to consider including reflective questions as part of the assessment their own learning 

of student learning. pore hee wr. Be 

ePortfolios. Increasing the use of ePortfolios can capture student learning over the entire provide them with 

ss i pc Ue EA cn acetates Cs a to nee opportun hes ie 
ples of their work, reflect on their progress towards personal and educational goals, 

and share their work with faculty, family, and future employers. ePortfolio practice was reflec t on their 

recently added to the list of High Impact Practices (HIPs) (Watson et al., 2016). HIPs Ioecie 

describe a wide range of college experiences that have been shown to increase rates of & 

student retention and engagement. ePortfolios are hypothesized to function as a meta- 

HIP by increasing the impact of other HIPs such as undergraduate research, learning 

communities, internships, writing-intensive classes, and study abroad (Kuh et al., 2018). 

ePortfolio practice has become increasingly common in institutions of higher education 

with over half of U.S. colleges reporting ePortfolio use to capture students’ learning (Eynon 

& Gambino, 2017). ePortfolio use is likely to increase as employers report that they find 

them to be more useful than resumes for hiring decisions (Hart Research Associates, 2018; 

Watson & McConnell, 2018). At the heart of ePortfolio practice is the documentation 

of and reflection on learning. Reflection on learning supports students’ self-evaluative 

abilities (Singer-Freeman & Bastone, 2019b). A high quality ePortfolio curates curricular 

and co-curricular experiences into a cohesive picture of a student’s education. As such, it 

has the potential to document the full range of learning that students experience across 

time and different institutions (Hundley, 2019; Kinzie & Jankowski, 2015). When 

ePortfolios are created over the full span of a student’s educational experiences they can 

be used to measure changes in learning over time. ePortfolios also stimulate and measure 

students’ awareness of their own learning (Singer-Freeman & Bastone, 2018; 2019a; 

2019b). Reflection is increasingly viewed as an essential element of higher education 

(Hutchings, 2018). The fact that ePortfolios encourage and document reflection positions 

them to be a significant tool for both assessment and learning improvement (Hutchings 

et al., 2015). We encourage institutions to increase the use of ePortfolios to provide 

students with documentation of their own learning over time and to provide them with 


opportunities to reflect on their learning. 
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Massive Data. It is time to consider leveraging technology to analyze massive data sets 
within and across institutions. In so doing, we may improve the quality of our data and 
the sophistication of our analyses to maximize the usefulness of assessment findings (Avella 
et al., 2016; Baer, 2017; Brown, 2017; Eubanks, 2019a; 2019b). Evolving technologies 
(e.g., MOOCs, course management systems, artificial intelligence, and adaptive learning 
platforms) now collect large data sets with a wide range of variables including individual 
assignment and problem grades, time spent on tasks, and use of linked resources. These 
data present institutions with opportunities to look in more nuanced ways at evidence of 
learning (Dirlam, 2017; Pena, 2018; Suskie, 2016), including disaggregation and effects of 
prior learning. Technology also could enable small institutions to conduct higher quality 
assessment by sharing data, data systems and assessment personnel (Ariovich et al., 2019). 
A crucial step towards analyses that span institutional boundaries will be consensus on what 
should be measured, what common metrics might be used, and how the resulting findings 
will be ethically and purposefully used (Banta et al., 2016; Lingenfelter, 2016). Competency- 
based education (CBE) might be one approach to the creation of uniform metrics that 
can be compared and transferred between institutions, however; there are currently many 
different approaches to measuring competencies. Lurie and Garrett (2017) describe current 
implementation of competency-based education stating “There is a tension between the 
logic of CBE, which pushes a more standardized approach to establishing competencies, 
curricula, and course content, and the decentralized culture of higher education” (p. 18). 
This tension captures the general concern among faculty that the desire to create uniformity 
in metrics may come at the cost of using more authentic forms of measurement of student 
learning (Braun, 2019). If we successfully address this challenge, we might begin to engage 
in “meta-assessment” by examining the relative efficacy of different assessment methods and 
exploring the extent to which assessment findings are driving improvements. We encourage 
assessment organizations and higher education organizations to consider ways in which they 
might contribute to cooperation around the determination of metrics and analyses that span 
institutional boundaries. 


Communicate. Finally, effectively communicating relevant, timely, and contextualized 
information about the full range of experiences to stakeholders will contribute to student 
learning and success (Hundley, 2019; Jankowski et al., 2018). With attention to the needs of 
different audiences, we must begin to offer clear information about why we provide students 
with different experiences in higher education and how those specific experiences benefit 
students (Jankowski & Cain, 2015; Jankowski & Marshall, 2017). The key to improving 
assessment is collecting information that informs decisions and effectively communicating 
this information to all stakeholders (Hutchings et al., 2015). One important element of 
effective communication might be to make incremental improvements that accumulate over 
time visible to students and other stakeholders (Stanny, 2018a; 201 8b). Reframing assessment 
as a process of “meaning making” will improve an institution's ability to communicate with 
stakeholders effectively (Penn, 2018; Rhodes, 2018; Watson & McConnell, 2018). The 
most recent strategic plan for the Association of American Colleges and Universities and a 
recent statement in NILOA Viewpoints (Higher Education Assessment Practitioners, 2018) 


call for an acceleration in advocacy and outreach. Clear communication is also needed 
internally. We must improve the extent to which we share meaningful assessment findings 
with students, faculty, and other stakeholders to accomplish meaningful improvements in 
response to assessment findings (Horst & Ames, 2018; Kuh & Ikenberry, 2018). Successful 
progress towards the resolution of any of the other challenges will require improvements in 
communication in order to identify solutions, generate funding, and coordinate progress 


towards goals. ; 
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Final Words 


Assessment in higher education has a range of challenges related to improving 

measurement, addressing inequities, and fostering continuous improvement. To enable 

collective solutions to these grand challenges, the shift away from compliance towards 

assessment for improvement needs to be fully realized. It is clear from the responses of 

assessment professionals to these challenges (Singer-Freeman & Robinson, 2020) that 

they are ready to use assessment to increase equity, innovation, pedagogy, and document 

progress over time. The concept of grand challenges is an effective one for the assessment [QQ 
community because we do have challenges, we cannot solve them alone, and we need the 

collective efforts of the assessment community in order to move the needle. We hope you Join us in C reatively 

will join us in creatively addressing these problems. addressin g the 


grand challenges of 


assessment. 
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